…ebugging requests"

Swift does not return all the parameters of objects in a listing
(e.g. ChecksumType and ChecksumAlgorithm) so pop these from listings
before making assertions.

Change-Id: Ieb7a9783731c11f1c08db398eae07ffafa127460

Change-Id: Iea1adfb93891e4bde62a11bfcef478d9b9696fd4

It used to be that when a logger encountered an error trying to handle
/ emit a message, it would attempt to log about the error via the same
logger. Long ago, this could lead to infinite recursion in Swift's
LoggerFileObject, so we fixed it and added a test that included
showing the bad stdlib behavior.

Recently, stdlib did a similar fix so the recursion doesn't happen. See

 - python/cpython#91555 and
 - python/cpython#131812

Of course, now our test breaks (at least, on 3.13.4). Relax the
assertion a little.

Related-Change: Ia9ecffc88ce43616977e141498e5ee404f2c29c4
Change-Id: Id2f490d4204b8eaf07857cb84ed783bec19b8511

Change-Id: I32fec7540bee70e77140964c5983d133a572fa7b

... because Python 2.x is no longer supported.

Change-Id: I3167a539b3e26ceb35976fbd7a2356ba59d4a5e4

Change-Id: I4eff901aaa334c8a73cebfc578cea14d23e6c365

datetime.timezone.utc[1] has been available in Python 3 and can be used
instead of datetime.UTC which is available only in Python >=3.11 .

[1] https://docs.python.org/3.13/library/datetime.html#datetime.timezone.utc

Change-Id: I92bc82a1b7e2bcb947376bc4d96fc603ad7d5b6c

Change-Id: I74381567978025e7f528e169692bbaf9bac45de2

The existing doc is quite outdated and is based on ancient versions.
Update it according to the following points, so that it works in recent
versions.

* Use python3- packages instead of python-/python2- packages
* xinetd is no longer available in recent CentOS
* Remove old unused test dependencies such as nose and mock
* Remove netifaces which is no longer a hard dependency

Change-Id: I8bf87f858406dc1a32139a0071b53cfb90864108

Change-Id: I80142c6c0654b65b5755e7e828bcc4969a10f4f1

Change-Id: I88e2e743ccbd6292bc1570ae0efbdd45dcced8cc

If we already know enough about the error to feel confident in
quarantining, we don't really need a whole traceback about it;
that just clutters logs.

Change-Id: Iab0e62c85b33d699f96d744faaa16420b7148b47

S3 includes the expected base64 digest in a BadDigest response to a
multipart complete POST request.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ie20ccf10846854f375c29be1b0b00b8eaacc9afa

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: I9a31eda60da71ff9f9db8b32b517338e99896667
Signed-off-by: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Generated-By: openstack/openstack-zuul-jobs:roles/prepare-zanata-client/files/common_translation_update.sh

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
for some background.

This covers both "normal" objects and part-uploads for MPUs. Note that
because we don't write down any client-provided checksums during
initiate-MPU calls, we can't do any verification during complete-MPU
calls.

crc64nvme checksums are not yet supported; clients attempting to use
them will get back 501s.

Adds crt as a boto3 extra to test-requirements. The extra lib provides
crc32c and crc64nvme checksum support in boto3.

Co-Authored-By: Ashwin Nair <ashnair@nvidia.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Id39fd71bc59875a5b88d1d012542136acf880019

Adds a test that verifies extra body content beyond the content-length
is ignored provided that the checksum value matches that of the
content-length bytes.

Add comment to explain why this is the case.

Drive-by: add clarifying comment to unit test.

Change-Id: I8f198298a817be47223e2f45fbc48a6f393b3bef
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Assert that BadDigest responses due to checksum mismatch do not include
the expected or computed values.

Change-Id: Iaffa02c3c02fa3bc6922f51ecf28a39f4b24ccf2
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

The py3 tag is preserved with multiple tags in the swift-*-image jobs.

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Ic1a5f5ed4ab5fe0fd278083a919efabf39f72c56

The author has observed this test fail because the request somehow
succeeded in connecting to 10.0.0.x addresses, meaning that node
timings *were* unexpectedly set. Even when the test behaves 'normally'
there is a >= 0.5 sec delay waiting for a connection timeout. This is
unnecessary: we can simply mock the connections to raise Timeouts
immediately.

Change-Id: Iff2f0e6dffca73eb689bf354e278e0f727f881af
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I53445f574953d9d47dd93d6819060e4b3cabee75

Signed-off-by: Matthew Oliver <matt@oliver.net.au>
Change-Id: Ieaf9ca9a67e40a7133f56f4fde86a88ba4666ac4

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I202ac65afd0fa1812cc7b5e0b6775489ac1daedd

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I5ad18f1ef0ef040fc65d34904fad2d324b451d0d

This commit adds support for barbican_region_name configuration option
in Swift's kms_keymaster middleware to enable proper multi-region Barbican
endpoint selection via Castellan.

Changes:
- Added 'barbican_region_name' to keymaster_opts tuple to allow reading
  the option from configuration
- Set barbican_region_name in Castellan's config after calling
  options.set_defaults() so it's available to BarbicanKeyManager
- This enables Castellan's BarbicanKeyManager to use the region for
  endpoint discovery in multi-region deployments

The barbican_region_name option should be set in the [kms_keymaster]
section of the keymaster configuration file (or proxy-server.conf).
Castellan's BarbicanKeyManager will use this value when discovering
the Barbican endpoint from the service catalog.

Related-Bug: #2138973
Signed-off-by: Ade Lee <alee@redhat.com>
Change-Id: I02e86c736398698d1bae19c17e9d97b1974e5054

…ware"

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I21c13b42f5d35b0ac14d71f3cc115bac12503aec

There's no need for the s3api controllers to set the x-timestamp
header in requests. Furthermore, it may be inappropriate for the s3api
middleware to set the x-timestamp header if in doing so it overwrites
a value set by a previous middleware.

In the case of an object copy, the last-modified time required for the
response body can be obtained from the swift copy (PUT) request
response.  The proxy server app copies the PUT request.timestamp to
the PUT response.last_modified.

Related-Change: Iab68f4b596dfa7f23bc587fbde859d854faff87f
Related-Change: I4dacc11ec086fdd2e9bb3f68295f2ebaea68bede
Change-Id: I4f46ae51579aafb9f511ddec8f2bce5fd5498ba5
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

S3 and Swift both support expiring objects in some form, but the
semantics are entirely different.

In S3, expiration policy is at the bucket level. A client calls
PutBucketLifecycleConfiguration to specify bucket lifecycle rules;
some of these rules enable object expiration. After that API call,
the rules take effect within a few hours for all objects in the bucket.

In Swift, expiration policy is at the object level. When uploading an
object, a client can add the header “X-Delete-At: T” or
“X-Delete-After: D”, where T is a Unix timestamp and D is a duration
in seconds. That object will then expire at either time T or at
D seconds after the PUT request was made.

S3 object GET and HEAD responses for expiring objects contain the time
at which the object will expire based on the current bucket configuration
as well as the rule responsible. This is in the “X-Amz-Expiration” header.
Example:
    x-amz-expiration: expiry-date="Fri, 16 Jan 2026 00:00:00 GMT",
                      rule-id="logs-expiration"

This patch adds this support into Swift’s S3API for expiring objects.
Swift's S3API does not currently support expiration via bucket lifecycle
policies, so the x-delete-at header is the only possible source of
expiration time to be returned in the x-amz-expiration header. However,
if bucket lifecycle expiration were to be implemented then the
x-amz-expiration header could return the lowest of the bucket expiration
time or the x-delete-at time.

Adds Sam, who came up with this approach, as co-author.

Change-Id: I9e319b75c557d5ab971182adb4aa10ad27a14d0b
Co-authored-by: Samuel Merritt <smerritt@nvidia.com>
Signed-off-by: Jianjian Huo <jhuo@nvidia.com>

Change-Id: If05f93a74f576b485118944a06a51b09f8fdd54a
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Previously, in object_versioning middleware, when writing a delete
marker to the versions container, the timestamp value used to
calculate the delete marker version id was different to the
x-timestamp given to the delete marker PUT sub-request. This was
inconsistent with the behaviour when writing a version to the versions
container, where the x-timestamp value of the PUT to the versions
container is the same as the value used to construct the version id.

With this patch, the same timestamp value is used for the delete
marker version id and the corresponding PUT sub-request.

The patch also adds tests to verify the x-timestamps that are applied
by the object_versioning middleware.

The FakeSwift test helper class is extended to support default
responses which will match any path for a given path. This allows
requests to be captured whose path may not be known a priori (for
example, the path to a new version object).

Change-Id: I1c2794bb7ac4eb984ae6b48f41ecafa76e9e7e4c
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Previously, requests would be assigned an x-timestamp header (if they
do not already have one*) by various means in the proxy account,
controller and object controllers. With this patch, all requests are
assigned an x-timestamp header at one place early in the proxy server
Application (i.e. the update_request method).

BaseController.generate_request_headers now copies an x-timestamp from
the original request, rather than setting x-timestamp.

Note: a request may already have been assigned an x-timestamp in
middleware (e.g. SLO, object versioning). Previously, the proxy
Application would only set an x-timestamp header if none existed (in
BaseController.generate_request_headers), *except* for container PUT,
UPDATE and DELETE requests which would have an existing x-timestamp
header replaced (in ContainerController._backend_requests).

With this patch the proxy Application never overwrites an existing
x-timestamp header.

Drive-by: The BaseController.generate_request_headers orig_req
argument is no longer optional. All existing call sites pass an
orig_req value.

Change-Id: Ic32d32f27157d1ba47e169ff1a22bce6f5a89395
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

It's not immediately obvious, but the direct client functions always
set an x-timestamp header in request headers if the header does not
already exist. This patch refactors the module so that the timestamp
is set in one place.

Drive-by: fix missing call to do_test helper in
test_direct_get_container_with_extra_params.

Change-Id: I7981525dc8d800aa9304fb5fb837e5361b5eeaa0
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

The author found the print output confusing while trying to sanity
check the output of the test's logger during review of the
Related-Change.

Related-Change: I046c4837a638d097649688db39455fb089fa7007
Change-Id: I81fed3bb02da33322804813636223949ec03fdc4
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Various parts of the relinker record the invoked action ('relink',
'cleanup') in generated log messages, without utilizing a uniform
format. In most cases, the logged output included cleanup=(True/False).

This patch improves this by using a prefixed logger that prepends log
messages with the invoked action's name, which seamlessly supports any
actions added to the relinker in the future.

Change-Id: I046c4837a638d097649688db39455fb089fa7007
Signed-off-by: Wael Halbawi <whalbawi@nvidia.com>

Jammy is not in testing runtime for this cycle and we can drop
this job which is also failing 100%.

Change-Id: Ia9e5782ee8f632951412898895a6134464e9dc3e
Signed-off-by: Ghanshyam Maan <gmaan.os14@gmail.com>

Strengthens the assertions (previously relaxed in [1]) in
TestRelinker::test_state_file. The test now asserts that the step
(relink/cleanup) is included in the logged output.

[1] I046c4837a638d097649688db39455fb089fa7007

Change-Id: I4449e8fded45b74db4463008c9839a33b48ffd17
Signed-off-by: Wael Halbawi <whalbawi@nvidia.com>

Previously, normalize_timestamp was widely used in tests to generate a
string representation of a Timestamp. However, tests should often be
using the internal format of a Timestamp. This patch retires the use
of normalize_timestamp in tests.

The preferred mechansim for generating sample timestamps in unit tests
is to use the timestamp iterator helper available via
BaseUnitTestCase.ts_iter and BaseUnitTestCase.ts(). This patch adopts
this pattern, but there is still further work required to make all
tests use the pattern consistently.

Change-Id: I2b6cee7e7a189294d4e28b7cb2aadadd10a11c2d
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

test_db was using its own self.ts which was a timestamp_iter whereas
other parts of swift unittests inherit their testcase from BaseUnitTestCase
which has `self.ts_iter` as the timestamp_iter and self.ts as a helper
method which calls `next(self.ts_iter)`.

Meaning this code although looking similar actually behaves quite
differently. So this patch changes test_db to use the BaseUnitTestCase
everywhere it uses `self.ts` to keep it consistent with other
tests in swift.

In summary, TestDbBase extends BaseUnitTestCase and next(self.ts)
becomes self.ts().

Change-Id: I4db72dc72a274838cbc4cd54e8ce17770eddfd0a
Signed-off-by: Matthew Oliver <matt@oliver.net.au>